Multi frame holograms batched optimization for binary phase spatial light modulators

Phase retrieval methods used in computer generated holograms such as Gerchberg-Saxton and gradient descent give results which are prone to noise and other defects. This work builds up on the idea of time-averaging multiple hologram frames, first introduced in methods like One-Step Phase-Retrieval and Adaptive One-Step Phase-Retrieval. The proposed technique called Multi-Frame Holograms Batched Optimization uses the L-BFGS optimization algorithm to simultaneously generate a batch of binary phase holograms which result in an average reconstructed image of improved fidelity and fast algorithmic convergence, both in the Fraunhoffer and the Fresnel regimes. The results are compared to One-Step Phase-Retrieval and Adaptive One-Step Phase-Retrieval in simulation and experimentally, proving the superiority of the proposed approach. This technique can be easily extended to other spatial modulation methods.


Method
This paper proposes a novel method of using numerical optimization algorithm L-BFGS 15 to generate multiframe binary-phase holograms.The L-BFGS algorithm had previously been used for single-frame hologram optimization 9,10 .To implement it onto multi-frame holograms generation, the argument to vary becomes the set of holograms with n sub-frames ( {H 1 , H 2 , . . ., H n } ), each having a resolution of X × Y pixels matching the reso- lution of the target image, and the objective function to minimise is therefore the difference between the average reconstruction amplitude R avg = 1 n n i=1 R i and the target image ( T ), which is denoted as Loss(T, R avg ) , where n is the total number of frames, R i 's are reconstructions from individual hologram sub-frames H i 's for i ∈ [1, n] .To compute each R i from the corresponding H i , we start from the Fresnel diffraction formula given in Eq. ( 1) 16 where E is the reconstructed electric field, in complex form, A is the hologram aperture, also in complex form, and F denotes the Fourier Transform, implemented on computers using the Fast Fourier Transform (FFT) function.As eyes cannot perceive phase, the reconstruction amplitude is therefore the absolute value, giving R = |E| .And as we are generating holograms for phase-only SLM's, A is then comprised of a uniform amplitude with phase H, giving A = e jH , where the exponential is taken element-wise.
To help explain the optimization process, a flow chart is drawn in Fig. 1.As shown in the flowchart, the target image T is first loaded, with a set of n hologram sub-frames ( {H 1 , H 2 , . . ., H n } ) generated randomly.Then at every iteration, each hologram sub-frame H i is quantized to the bit-depth constraint of the SLM, and propagated to the reconstruction plane R i , and the average of the amplitudes of all reconstructions R avg is computed and compared against the target image T using a loss function Loss(T, R avg ) , after which the search direction is computed using the L-BFGS optimizer and the hologram sub-frames are updated accordingly.Here the loss function selected is the relative entropy 17 given in Eq. ( 2).
Since fast SLM's available in the lab are binary-phase devices, the quantization step in the flowchart in Fig. 1 is carried out with bit-depth limit of 1, hence producing binary-phase holograms.However, the optimization algorithm does not converge with a straight binary quantization as integers are discrete, therefore a Sigmoid function 18 is used for a smoother and differentiable quantization, as defined in Eq. ( 3).The output of the Sigmoid function is then scaled by π so that the binary phase levels are 0 and π.
And finally, when displaying the multi-frame holograms, each of the n frames generated are then rounded to binary phase values and displayed on the binary phase SLM sequentially.And when the first round finished, the second round starts with the first frame again (i.e. after frame n, the next frame displayed is frame 1), and such infinite loop doesn't stop until another set of holograms are uploaded.

Simulation results
To test the proposed MFHBO method, a target image T as shown in Fig. 2 was used.It was designed from the widely used mandrill image 19 .A rotational symmetry was introduced to match the rotational symmetric property of the far field projections from binary phase holograms.It was then zero padded to a resolution of 1024 px × 1024 px and subsequently interpolated to a resolution of 1280 px × 1024 px to match the resolution of the SLM in our lab.Note that the target image was zero padded to a square aspect ratio and then stretched to the non-square aspect ratio because more pixels in the horizontal axis only means higher sampling rate as part of the features of the FFT, the replay field is continuous and is not pixelated and the simulated reconstruction of 1280 px × 1024 px resolution is the sampled results, which will be illustrated visually in Fig. 5 later.
To further explain the optimization process described in Fig. 1, an example iteration with n = 24 is shown in Fig. 2. At each iteration, every hologram is quantized and propagated to the reconstruction plane, forming {R 1 , R 2 , . . ., R 24 } .The average reconstruction amplitude R avg is then compared against the target image T , using www.nature.com/scientificreports/ the loss function in Eq. ( 2).The holograms {H 1 , H 2 , . . ., H 24 } are then updated according to the search direction calculated using the L-BFGS optimizer.After setting the optimization to terminate when the number of iterations reach 1000, the same algorithm was run on the same target for different number of frames (n), the normalised mean squared error (NMSE) and the peak signal-to-noise ratio (PSNR) between the average reconstructions R avg and the target image T were calculated at every iteration and plotted in Fig. 3a,b respectively.The plots in Fig. 3 show that the proposed MFHBO method has achieved good convergence within 400 iterations, for the various number of frame settings n in {1, 2, 3, 4, 6, 8, 12, 24} .The final NMSE values in Fig. 3a are difficult to distinguish in the plot, therefore it will be further compared in the bar chart in Fig. 5.The number of frames are chosen to be integer factors of 24, which is determined by our experimental setup, further explained in the next subsection.
The programme runtime of the proposed MFHBO method has been measured on a laptop computer of model ASUS ROG Zephyrus M16 (GU603H) with a CPU of model i7-11800H and a GPU of model NVIDIA RTX3060 and the results for different combinations of number of frames and number of iterations are listed in Table 1.It can be concluded that the application of the proposed method is for pre-computed high-quality holograms, instead of real-time holographic projections.

Optical experiment results
The holographic projection system used in this experiment is the same as the one used in previous research 21 , which was originally developed by Freeman 20 .The optical setup is shown in Fig. 4. The design is consisted of a diode-pumped solid-state (DPSS) 532 nm 50 mW laser source, focused down by an aspheric singlet, and passed through a polarising beam splitter cube to a collimating lens, which illuminates the SLM 20 .The SLM is a binary  Since the SLM has a refresh rate of 1440 Hz and modern computer monitors have refresh rate of at least 60 Hz, the maximum number of frames was chosen to be 1440/60 = 24 , so that each set of 24 frames will take a total of 1/60 s to display, therefore giving an equiva- lent refresh rate of 60 Hz.Then the integer factors of 24 were chosen so that the equivalent refresh rate becomes integer multiples of 60 Hz.The number of frames starts from 1 to help illustrate how the increase in number of frames positively affect the reconstruction quality.
The results in Fig. 5 further compares the final results for different number of frames.The histogram in Fig. 5 shows that, as the number of frames increases, the NMSE between the average reconstructions R avg and the target image T decreases and the structural similarity index (SSIM) 22 increases, showing a trend of better reconstruction quality with higher number of frames.Such trend is expected as more frames provide higher information capacity, which agrees with the previous research where holograms with higher bit depth were found to achieve better reconstruction quality 23 .The trend is also shown visually via the simulation results and their detail enlargements.The corresponding multi-frame holograms are then loaded onto the SLM, and the reconstructed field is captured using a camera of model Cannon EOS 1000D.Only the bottom halves of the reconstructed field were captured as the symmetrical conjugates were unwanted feature of far field projections from binary-phase SLM's.The raw data including multi-frame binary-phase holograms, simulated reconstructions and optical results captured are accessible in the database 24 .
Then another target image was tested, which is the holography ambigram shown as shown in Fig. 6 (Adapted, with colours reversed, from holography -Benjamin Wetherfield, 2022).The term ambigram is used to refer to (often typographical) designs that are invariant under a reflection, rotation or other symmetry.The 'holography' design contains 180° rotational symmetry, which makes it especially well suited to binary Fourier-holographic projection, where this symmetry is unavoidable.Multi-frame holograms were then generated using the proposed MFHBO method and the existing OSPR and AD-OSPR methods, for the same number of frames n = 24 .And the optical results are shown in Fig. 7.As shown in Fig. 7, for the Mandrill target image, it can be seen that the proposed MFHBO method achieved a much better optical reconstruction quality than the existing OSPR and AD-OSPR methods, with clearer details and better contrasts; for the 'holography' ambigram target image, the proposed MFHBO method is shown to have a much lower background noise around the centre, than the existing OSPR and AD-OSPR methods.The intended black regions are represented much more cleanly, with an elimination of speckle-like artefacts in the zero-valued space around the lettering, and an overall increase in discernible contrast.
A quantitative analysis was then conducted on the optical results in Fig. 7, the NMSE and SSIM between the captured reconstructions and there corresponding targets are computed and listed in Table 2.The NMSE results of the proposed MFHBO method are lower than those of the existing OSPR and AD-OSPR methods, with a 25% reduction on average among both target images.On the other hand, the SSIM results have shown a 62% increase using MFHBO than OSPR and AD-OSPR for the mandrill target image, but a slight decrease of 3.7% for the 'holography' ambigram target image, which is negligible as it is less than 5% and the SSIM metric is not originally designed for binary-valued non-grayscale images.Table 2. Quantitative analysis of the optical results in Fig. 7.

3D holography
The proposed MFHBO method was extended to multi-slice targets, by computing the loss between all 4 slices of reconstructions and target images (the Sum-of-Loss method in 10 ).An example 4-slice target made from alphabets ' A, B, C, D' is shown in Fig. 8.The z values, corresponded to the z variable in Eq. ( 1), were chosen to be 1.1, 1.9, 3.5, 7.7 for the 4 slices respectively (as there's no correlation between each slice, larger separation was chosen for fewer cross-talks across different planes).It can be seen that the proposed MFHBO method has produced sharper edges in reconstructions than the existing OSPR method.(The AD-OSPR method was not attempted here as its application to multi-slice targets was not defined).Then a quantitative analysis was carried out, with NMSE and SSIM values measured and shown in Table 3.The proposed MFHBO method has shown a 10% reduction in NMSE and a 8% improvement in SSIM on average than the existing OSPR method, demonstrating the effectiveness of the proposed method.
Lastly, a set of real-life scene was captured in the lab using near, middle and far focus, as shown in T 1 , T 2 , T 3 in Fig. 9 respectively.The z values were set to 1.1, 1.2, 1.3 for hologram generation, and the reconstruction results of the existing OSPR and the proposed MFHBO methods are compared in Fig. 9.The proposed MFHBO method is shown to have achieved much better reconstruction quality than the existing OSPR method.
A quantitative analysis was conducted again, with NMSE and SSIM values measured and listed in Table 4.The proposed MFHBO method has shown a 17% reduction in NMSE and a 7% improvement in SSIM on average than the existing OSPR method, proving the effectiveness of the proposed method.

Conclusion
This paper proposed the MFHBO method to generate multi-frame binary-phase holograms to be displayed on high refresh rate binary-phase SLM.The proposed MFHBO method was shown to achieve much better reconstruction quality and higher contrast than the existing multi-frame binary-phase holograms generation methods OSPR 13 and AD-OSPR 14 on the holographic projector with binary-phase SLM, for all the single-slice far-field targets and the multi-slice near-field targets tested.Although the propose MFHBO method is slower than the existing OSPR and AD-OSPR methods, its much better reconstruction quality makes it suitable for pre-computed high-quality hologram applications.Its strong advantage for high contrast target such as the 'holography' ambigram, with much suppressed speckle noise in the background, makes it well-suited for photo-lithography applications.The proposed method can also be adapted for multi-level SLM's by simply removing the quantization step (in Fig. 1).This could be the case for applications such as photo-lithography, where the time response of the system is much longer than it is for human vision, and the high refresh rates of the SLM are not necessary.

Data availibility
The datasets generated and/or analysed during the current study are available in the Apollo -University of Cambridge repository, https:// doi.org/ 10. 17863/ CAM.109607.

Figure 2 .
Figure 2.An example iteration in the optimization process.

Figure 7 .
Figure 7. Optical results comparison of the proposed MFHBO method against the existing OSPR and AD-OSPR methods.

Figure 9 .
Figure 9. Real-life captured image as target field and their reconstruction results.

Table 3 .
Quantitative analysis of the optical results in Fig.8.

Table 4 .
Quantitative analysis of the optical results in Fig.9.